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Abstract. We prove a formula of Petersson's type for Fourier coefficients of Siegel cusp 
' forms of degree 2 with respect to congruence subgroups, and as a corollary, show upper 

04 . bound estimates of individual Fourier coefficient. The method in this paper is essentially 

^ ' a generalization of Kitaoka's previous work which studied the full modular case, but some 

O . modification is necessary to obtain estimates which are sharp with respect to the level aspect. 

o 

(N 



1. Introduction and the statement of main results 
Let Mi{R) be the set of / x / matrices whose components belong to a ring R, and define 



^' : A = {Se M2(Z) I 'S = S}, 

^ : A* = {S= (si,) G M2(Q) I su G Z, 2su = ^s^i G Z}. 

Let U2 = [Z = X + iY e M2{C)\^Z = Z,Y > 0} be the Siegel upper half space of degree 2, 
Sp(2, Z) be the Siegel modular group of degree 2 which consists of all non-singular matrices 

^ M G M4(Z) satisfying *MJM = J, where J = ^ . For a positive integer iV, define 



00 

^f\N) = {{^ f) GSp(2,Z) 



C = ( Q Q 1 modiV 



(1') 

Denote by ^fc(F^"^(A^)) the space of Siegel cusp forms of weight k{> 1) with respect to 
Fq (A^), and write the Fourier expansion of / G 5'fc(Fg (A^)) as 

(1-1) f{Z)= a/(Q)e(tr(gZ)), 

where Z G H2, e(x) = exp(27rix) and tr denotes the trace map. The main purpose of the 
present paper is to prove a Petersson type formula for the above Fourier coefficients af{Q). 
Such a study was carried out by Kitaoka [5] in the full modular case, in order to obtain 
an upper bound estimate of Fourier coefficients. Our motivation is to generalize Kitaoka's 
result to the case of general Fo(iV). 

First recall the classical elliptic modular case. Let 5'fc(Fo(A^)) be the space of elliptic cusp 
forms of weight k and level N, f & Sk{TQ{N)), and denote its Fourier coefficients by af{n). 
Then the classical Petersson formula is 



r(fc-l) ^af{m)af{n) 



(47rv/^)^-i^ (/,/) 



00 

c=0 (modN) 



Key words and phrases. Siegel modular form, Fourier coefficients, Petersson formula. 
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where ( , ) denotes the usual Petersson inner product, 5mn is the Kronecker delta, S{m, n; c) 
is the Kloosterman sum, Jfc„i(-) is the {k — l)-th J-Bessel function, and the sum on the 
left-hand side runs over an orthogonal basis of Sk{N) (see Theorem 3.6 in [3]). From this 
formula, evaluating the sum on the right-hand side, we can show 

where d{-) denotes the divisor function (Duke [1], Kamiya [4]). 

(2) 

Now return to the Siegel case. Let J^k,N be a set of orthogonal basis of 5'A,.(rQ (A^)). For 
Q, T e A*, define 

5{Q, T) = i^{U E GL(2, Z)\UQ'U = T}. 

In what follows, Q is to be regarded as fixed, and e denotes an arbitrarily small positive 
number, not necessarily the same at each occurrence. The constants implied by Landau's 
0-symbol and Vinogradov's ^ symbol may depend on Q, e. 
Now we state our main results in the present paper. 

Theorem 1.1. Let Q, T G A*, both are positive definite. Then 

(i) We have 

(1.4) 7rV2(47r)3-2'^r(A; -|)r(fc-2)(det Q)-^+t ""^^^J^^ 

= 5{Q,T) + Eq{T,N), 

where Eq{T,N) is the error term, in the sense that it tends to when N — > oo. Moreover 
the estimates 

(ii) Eq{T, N) = 0{nI-''\T\^-I + N^-''+'\T\^--4+' + Ar3-2fc+e|T|'=-i+^), 

(iii) Eq{T,N) = o(Ar-V2+e|r|/c/2-i/4+.^ 

hold for k > 3. 

Remark 1.2. (i) When there is no f/ G GL(2, Z) satisfying UQV = T, obviously 6{Q, T) = 0. 
Therefore the role of S{Q, T) is similar to the delta symbol in formula (11. 3p . 

(ii) This result is a generalization of Proposition 3.3 in Kowalski-Saha-Tsimerman [7]. 
They applied the estimate to show an equidistribution result for L-functions associated to 
Siegel cusp forms of genus 2 and growing weight k. So it is expected that our result can be 
used to prove a similar result for growing level N. 

From the above theorem, as we will see in the next section, we can deduce an upper bound 
estimate of individual Fourier coefficient. 

Theorem 1.3. When there is no U G GL(2,Z) satisfying UQ^U = T, we obtain 

(a) af{T) = 0{nI-''\T\''-I + iV2-fc+.|2-||-i+e + j^3-2k+s\r^\k-i+sy 

(b) a/(T) = 0(Ar-i/2+^|T|'=/2-i/4+e) 
for k > 3. 

When = 1, Theorem 11.31 (b) is exactly Kitaoka's estimate [5]. However, the estimate 
with respect to is rather weak in (b). This point is supplied by (a), which gives a sharp 
estimate with respect to N. This (a) corresponds to the error estimate of Duke-Kamiya in 

In the following sections we will give the proof of the above theorems. Many parts of the 
proof are rather straightforward generalizations of Kitaoka's argument in [5] , but we describe 
the details because we have to trace carefully how is the effect of A^. In particular, some 
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modification of Kitaoka's argument is necessary to obtain estimates wliicli are sliarp witli 
respect to A^. 



2. POINCARE SERIES 

For M = (^^ ^ e Sp(2, Z) and Z e M2, we set 

j(M, Z) = det{CZ + D) 

and 

M{Z) = {AZ + B){CZ + D)-^. 



Moreover, we set 



■p(2)/ \ J A2 "S* 

Y\ '{00) =r- ^ ' 



1 



5 G A 



For Q G A* witli Q > and positive integers /c, A^, we define tlie Poincare series gisf{Z,Q) 
of weiglit k witli respect to rQ^^(A^) by 



g^iZ, Q)= ■ M{Z)))j{M, Z)-K 

Aferf (oo)\r(/'(7V) 

For fjQ^ Sk(rQ {N))^ we define the (unnormalized) Petersson norm of / and g by 

{f,g)= [ f{Z)'^){detY)'-'dZ. 

Proposition 2.1. Let f e S'fc(r{)^^(A^)). Then we have 

(2.1) {g^i; Q)J) = n'/'iAn f~'''T{k - ^)T{k - 2)(det Q)-'^l^), 

and consequently, 



(2.2) g^{Z,Q) = 7.y\4nr^''nk-^-nk-2)idetQ)-'^l ^^^tP' 

Proof. Tliis is a direct generalization of a result in Klingen's book [6], page 90. We briefly 
outline the argument. We follow the argument in pp. 76-90 of Klingen [6] with replacing 
r„ and An by Fg (A^) and F^ (00), respectively. The starting point, Proposition 1 (P 
p. 76]), does not depend on A^. Formulas (7), (8) in [6, p.78] are proved by the technique 
of decomposing the Siegel half space Hn into copies (by the action of {±1}\F„) of the 
fundamental domain F„. The same technique can be applied to our present situation, with 
replacing by Sn,kiN). In this way, we follow Khngen's argument until Proposition 3 ([6l 
p. 85]). In the statement of Proposition 3, the series G^iz; gi,) is deflned, but this is again 
independent of A^. (But be careful with the deflnition of A„.) Also A^ does not appear in the 
Fourier expansion of / G Sn,k{N). On the last line of p. 87, Klingen deflnes An, which differs 
from our T\ (00) by the factor 2. Therefore, g^{z,t) defined on p. 90 of [B] differs from our 
gN{Z, T) by the factor 2. All other parts of the proof are the same as in [6]. □ 
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Substituting (11. ip (with replacing Q by T) into the right-hand side of (12.21) . we have 
(2.3) g^(Z, Q) = 7r"=(47r)'-^'T(* - ^)T(k - 2)(det Q)-»+i 



Therefore, if we write the Fourier expansion of the Poincare series as 
(2.4) 9n{Z,Q)= AQ,iv(T)e(tr(TZ)), 

0<TeA* 

we obtain 



(2.5) AQ,^(T) = vrV2(4vr)^-^r(A:-^)r(fc-2)(detQ)-^+f ^^^JJjy^- 

Therefore the Fourier coefficient Aq^m{T) can be estimated by our Theorem 11.11 In partic- 
ular, when there is no [/ G GL(2,Z) satisfying UQ^U = T, from Theorem ll.il (ii). (iii) (with 
noting Remark ll.2p we find that Aq^isfiT) satisfies the estimations stated in Theorem 11.31 
Since any cusp form can be written as a hnear combination of Poincare series, we obtain the 
assertion of Theorem 11.31 

On the other hand, (12. 5p imphes that, in order to prove Theorem II. H it is enough to 
consider Aq n(T). 

(2) (2) (2^ 

Let i^jv be a complete system of representatives of P-^ (oo)\Pq {N)/T\ (oo). For M G 
Sp(2,Z), we denote 



9{M) = is eA 



Lemma 2.2 (Kitaoka [5], p. 158, Lemma 1). For M G Sp(2,Z), we have 
Pf)(oo)MPf)(oo)= H Tf\oo)M(^' f 

S€A/e{AI) ^ 

From this lemma, it is easy to see that 

(2.6) gN{Z,Q)= J2 Hq{M,S), 

M e N 

where 

(2.7) Hq{M,Z)= Y1 e{tT{Q-M{Z + S)))j{M,Z + Sy 

seA/e{M) 

Write the Fourier expansion of Hq{M, Z) as 

(2.8) Hq{M,Z)= J2 hQ{M,T)e{ii{TZ)). 

o<rgA* 

Then 

(2.9) hQ{M,T)= [ HQ{M,Z)e{-tT{TZ))dX, 

Jx{mod 1) 

where X = ( "''^ ] = 3ft(Z), dX = dxidx2dx4. 

\X2 Xi) 
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Comparing I^M>, <0> with (EH), we obtain 
(2.10) AQ,^iT)= J2 hQ{M,T). 



Therefore, to prove Theorem ll.H our remaining task is to evaluate each term on the right- 
hand side of (1210|) . Let 



= {M=(^ ^e^N\ rankC = i} 



for i = 0, 1 or 2, and decompose f l2.10p as 

(2.11) Aq,^(T) = So + Si + S2, 

where 

S, = J] hQ{M,T) (2 = 0,1,2). 

In the following three sections we evaluate Sq, Si and S2, respectively. 

3. The case of rankC = 
In this section, we assume rankC = 0, i.e. C = 0. 
Lemma 3.1 (Kitaoka [5j, p. 158, Lemma 3). ^45 S)^^ we can choose 

{('0 U-^ I ^eGL(2,Z) 

and e{M) = A. 
Proposition 3.2. We have 

So = #{f/ G GL(2, Z) I UQ'U = T}, 
which is hence non-zero only if Q ^ T. 



Proof. We can choose M which is of the form stated in Lemma [3.11 and 6{M) = A. Hence 
from (12. 7p we have 

Hq{M, Z) = e(tr(Q ■ M{Z)))j{M, Zf = e(tr(Q ■ 'UZU)). 

Therefore (12.91) gives 



/iq(M, T)= f e(tr(g ■ 'UZU))e{- ii{TZ))dX. 

Jx mod 1 



Then we have 



J2 hQ{M,T)= J2 I e{ti{Q ■VZU))e{-ti{TZ))dX 

X] / / / e{ti{Q -VZU -TZ))dxidx2dxi, 

r'l nr\ JO JO JO 



(7eGL(2,Z) 

where 

X = = mz). 

\X2 X4J 

Since tr(Q*[/Zf/ -TZ) = is equivalent to tr(([/g*[/ - T)Z) = 0, we see that if UQV = T 
then ii{Q^UZU — TZ) = for all Z, hence the above integral is equal to 1. On the other 
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hand, if UQ^U ^ T, then ti{Q^UZU — TZ) ^ for almost all Z, so the above integral 
vanishes. This completes the proof. □ 

4. The case of rankC = 1 
Next we consider the case rankC = 1. 



Lemma 4.1. ^4^ we can choose 



M 



U^^C'^V U^^D'V^^ ^ ^ Sp(2,Z) 



U eGi.V e G2,ci > 1,^4 = ±1, 
(ci,(ii) = 1, c?!, modci 



where 
C' = 



Nci 




D' 



di d2 




Moreover 6{M) is given by 



S eA 



* 



S[V] 



/GL(2,Z),G'2 = GL(2,Z) 





* 



1 * 

* 



for the above specialized M , where S\V] = ^VSV . 



When = 1, this is Kitaoka's Lemma 4 ([5], p. 159). The above generalization is obvious. 
For U, V in the setting of Lemma 14. 1^ we set 



and 



S 



Si S2/2 
^82/2 S4 

We choose an ai satisfying aiiii = 1 mod Ci. The following is Kitaoka's Lemma 1 ([3], 
p.l60) when = 1. 



Lemma 4.2. With the notation as above, we have 



hQ{M,T) = {-l)'/'V27f\Q\-^~^T\^-~-^6p,,s,s,HNci)-l 

( 



X e 



f aiSidl - {aid4P2 - S2)c?2 _^ O-lPl + ^ d4P2S2 



Nci 



Nci 



2NciSi 



NciSi J 



(4.1) 



Using this lemma, we evaluate Ei. First recall 



0((a, c)2c2). 



n mod c 

This is a well-known estimate on generalized quadratic Gauss sums, but here we give a sketch 
of proof. Denote the left-hand side by G{a, b, c). When (a, c) > 1, then G{a, b,c) = unless 
{a,c)\b, while in the latter case 

G{a, b, c) = (a, c)G{a/ (a, c), b/ (a, c), c/ (a, c)), 

so we may reduce the problem to the case (a, c) = 1. When (a, c) = 1, we write c = uv, where 
M is a power of 2 and v is odd. The decomposition G{a,b,c) = G{au,b,v)G{av,b,u) holds, 
and G{au,b,v) is explicitly written in the form rjv^, where 77 is a certain complex number 
with \ri\ = 1. Applying Theorem 10.1 of Hua pi Chapter 7], we find G{av,b,u) = 0(n2) 
(here, the e-factor in Hua's statement is not necessary because now u has only one prime 
divisor). Therefore G{a,b,c) = 0(c2) as desired. 



FOURIER COEFFICIENTS OF SIEGEL MODULAR FORMS 

Using (14. ip . we find 



E 



Nci 



0{{a,s,,Nc,)HNc,)^) = Oi{s,,Nci)HNci)r 



d2 mod Nci 

where the last equahty follows from the fact {ai,Nci) = 1. Therefore 



d2 mod Nci 



'^Sp,^s,\T\"-^s^^Nci)-\s4,Nci)-^ 



47rvigp 



(because Q is fixed and so the Q-factor is to be included in the implied constant), which 
further implies 



(4,2) 



E E 

C/eGl dimodJVci 

(dl,]Vci) = l,d4 = ±l 



E hQ{M,T) 



d2 mod Nc\ 



NC1S4 



Since Gi is parametrized by the second row up to sign, we see that the right-hand side of 
T2D is 



(4.3) 



< J2 Sp,,s,\T\-2-hJ{s4,Nci)^^ 



where M3, M4 is determined by U 



Ui U2 
Us M4 



A^CiS4 

. Since P = Q[^U], we have 



(4.4) 



Q[u] = (m3 U4,)Q 



Us 



Pa- 



Therefore 5p^^si = SQ[u],si, but Q[u] = S4} = 0{sl). Hence (US]) is 



S4, A^ci)2 



4-1 



'An,/\f\\Q\ 



NC1S4 



Therefore 
(4.5) 



EE E 



VGG2 C/eGl dimodJVci 

(di,JVci) = l,d4 = ±l 



E ''e(A'.r) 



« E i^-i 



c(2 mod Nci 
2 2 (S4,A^Cl)2 



47rv^gp 

A^CiS4 



We see easily that V is parametrized by the first column, and S4 = T ^ for 



^1 ^^2 



Therefore, setting 



A{m,T) = M';']\{v^,V2) = l,T 



V2 



m}, 
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we find that the right-hand side of (14.51) is 

'47r,/\T\\Q\ 



(4.6) 



< |T|2-4 ^ 



[s4,iVci)2A(s4,T) 



S4 = l 



Ju 



NciS4 



Here we quote the following well-known estimates: 

2 [ (n) (^(x 2) 

for X > (see Kitaoka [5], p. 163, Lemma 2). Applying 



(4.7) 



fi), we see that fl4.6l) is 



[s4,Arci)5A(s4,T) 



S4 = l 
00 



NciS4^ 



< iTl'^-i ^ A(s4,T)s4^+'4(iVciS4) 



S4=l 



Finally, since ^ < +00 (if > |), we have 



Cl = l yGG2 C/GGl dl modJVci 

{dl,JVci) = l,d4 = = 

00 00 
Cl = l S4 = l 

00 

< |T|'=-iAr-'=+i ^A(s4,T)s, 



S4 = l 

Since y4(s4,T) = 0(s|) (independent of T), we now arrive at the following proposition. 
Proposition 4.3. Ifk>3, then 

This proposition is necessary for the proof of assertion (ii) of Theorem II. 1[ To prove 
assertion (iii) of Theorem II. ![ we have to modify the above argument, using the both es- 
timates of (14. 7p . That is, to evaluate the Bessel factor in (14.61) . now we apply (14. 7p (ii) if 
A7f^/\T\\Q\ > NciSi, and apply (i) if 47r^|T||Q| < NciS^. Then (jM]) is 



«|T|"4(5i + 52), 



where 



E 

S4<t/Nci 

E 



S2 = 

S4>t/Nci 

with r = 47r-\/|T| Therefore 
(4.8) 



S4,iVci)2A(s4,T)|r|-4(iVciS4)^ 



S4,iVci)tA(s4,T)|T|t-t(iVciS4)-'+^ 



Si<|T|t-t^(5i + ^2) 



ci=l 
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Lemma 4.4. We have 



(4.9) Yl is^,Nci)HNc^)-2 <t:N--2+' (-^ 



2 

£ 

4) 



Sa 



l<AfCl<T/s4 



-j si {k>3). 

Proof. When = 1, this is Kitaoka's Lemma 2 ([5j, p. 163). As for (14. 9p . first we write 
(s4, Nci) = r and Nci = rq to obtain 

1<Nci<t/s4 r\s4 q<T/s4r,N\qr 

Put {N,r) = u, and write N = uN'. Then N\qr imphes N'\q, so we can write q = N'q'. 
Therefore the above is 

= E W)^ 

r\s4 q' <T / s^rN' 

-(-V 



— I 



< 



r|s4 

v\N r|s4,r=0(mod u) 
3 3 



where cri/2(A^) = '^d\N^^^'^- The proof of (14.101) is similar, but the condition A; > 3 is 
necessary in the course of the proof to assure the convergence of a relevant series. □ 

Using (jiyoj) and A(s4,T) < s|, we have 



ci=l S4<r/N ci<t/s4,N 



S4<t/N ^ 



2 

si 
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Also, using f liTTOj) . 

oo oo 

^52«|T|t-| J2 {s„Nc,)HNc,)I-' 

Cl = l S4=l Ci>t/s4N 

«|T|t-iiV-5+-^S4-^'+i- 

S4 = l 

Therefore, from (14. 8p we obtain 
Proposition 4.5. Ifk>3, then 



S4 



5. The case of rankC = 2 

The basic fact for the case rank C = 2 is the following lemma, which is Kitaoka's Lemma 
5 ([5], p.159) when = 1. 

Lemma 5.1. ^45 we can choose 



M 



NC 



eSp(2,Z) 



\C\ ^ 0,DmodA^CA 



ande{M) = {0}. 



The condition rankC = 2 is equivalent to \C\ ^ 0. For the set of such matrices, Kitaoka 
proved: 

Lemma 5.2 (Kitaoka [5], p. 164, Lemma 1). 
{C G M2(Z) I \C\ ^ 0} 

= { f/-Miag(ci, C2)V-^ I U E GL(2, Z), V E GL(2, Z)/P(c2/ci), < Cilcs} 
'ci 0' 



where diag(ci, 02] 



C2 



and 



a b 

c 



gGL(2,Z) 



6 = mod n 



The starting point of the argument is another formula of Kitaoka, stated in p. 166 of [5], 
which is 



hQ{M,T) =2~\ 



l_-4 



fe 3 

\T\ \ 2 ~4 



\Q\ 



\\NC\\-h{tT{AC-^Q + C-^DT)/N) 



„i 2 

"^0 j=i 



where si and S2 are positive numbers such that s\ and are the eigenvalues of the ma- 
trix Pn = T ^Q[\NC)-^] . (The symbol ||ArC|| means simply the absolute value of the 
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determinant |A^C|.) Then 

^ \\NC\\-iK{Q,T-NC) 

1^1/ 

„1 2 

1=1 

where 

X(g, T; C) = 5^ e(tr(AC-iQ + C~^DT)), 

D 

with D running over 



D mod A^CA 

,1 2 

X 



^ G Sp(2, Z) 



D(modCA) G M2(Z) 

This K{Q, T; C) is a kind of generahzed Kloosterman's sum, introduced and studied by 
Kitaoka [5] . In particular, Kitaoka proved: 

Lemma 5.3 (Kitaoka [5J, p. 150, Proposition 1). Let C G M2(Z) such that \C\ ^ and 



c = u-^ 



(o ca) ^' ^ ^ ° ^ "^^^^ -^"^^ P,T eA* we have 



K{P,T;C) = 0{clcf\c2,t,)-^), 

where e is any positive number and is the {2,2)-entry ofT[V]. Moreover, K{P,T] C) 
K{T,P-X) holds. 

By this lemma, we find that the right-hand side of (15.11) is 

,1 2 

(5.2) < \T\-2-i\\NC\\--^{Ncif{Nc2)'^+'{Nc2M)~^ 



u)u{\ — u ) ^du 



(because Q is fixed). 

Kitaoka ([5j, p. 166) showed that 

,1 2 

(5.3) 



^1 2 r(a) |P|t-t, 

-1 ' Lc) ipit-ti 



■(tr(P)) 



2 . 



where P = T ■ Q[\NC)~^]. Kitaoka stated the above (a) in case tr(P) < 1, (b) in case 
tr(P) < 2|P|, and (c) otherwise (which are sufficient for his purpose), but actually the above 
estimates themselves are valid without such conditions. This is because the estimates (14. 7p 
are true for any x > 0. In fact, applying (14. 7p (i) to the both Bessel factors of the left-hand 
side of (15. 3p . and noting (siSa)^ = |Po| = |P|, we obtain the estimate (a). Applying (14. 7p (ii) 
to the both Bessel factors we obtain (b). Applying (14. 7p (i) to the Bessel factor with smaller 
eigenvalue Si, and applying (14. 7p (ii) to the other Bessel factor, we obtain (c). 

We first use only (a) and (c) of (15. Sp to obtain an estimate which is sharp with respect 
to A^. It is possible to find a suitable Ui G GL(2,Z) for which A = T[Vdiag(ci, C2)^^f/i] is 
Mikowski-reduced. We may write C in Lemma 15.21 as 

C = U~^U^^diag{ci,C2)V-\ 

Then we have 

|P| = |T ■ Q[XNC)-^]\ = N~^\Q\ ■ \A\ X N-^\A\ 
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and 

tr P = tr(r ■ QIXNC)-']) X tr(T ■ hWNC)-^]) = N-^tT{A[U]). 
Therefore from fl5.2p we have 
(5.4) 



E 



(7ggl(2,: 



E hQ{M,T) 



D mod NCA 



l-k 
2 



!7eGL(2,Z),tr(A[[/])<l 

+ {N-^\A\)^--^ + Yl {N-^\A\)^-Hn-Ht{A[U])) 

(7GGL(2,Z),tr(yl[C/])<|A| {/eGL(2,Z),othcrwise 

where we apphed (15. 3p (a) to the first and third sums, and (c) to the second sum. 
Kitaoka proved ([5j, p. 167, Lemma 2) that if A is Minkowski- reduced, then 

(5.5) 

J2 \Af^--^+ Y \^\~~'+ lAl-^-hiiAp])'-^ 

C/6GL(2,Z),tr(yl[C/])<l UeGL{2,Z),tr{A[U])^\A\ l/gGL(2,Z),otherwise 

< m(v4)^max(l, |v4|)^+^|A|t"t-^, 

where m{A) = min{yl[x] \x G Z^, x ^ (0, 0)}. Using (15. 5p . we find that the first and the third 
sums on the right-hand side of ( 15. 4p are 

< Ar-^+2^(A)^ max(l, |A|)^+^|A|t-t-^ 

On the other hand, in the proof of the above Lemma 2 of Kitaoka, he proved (|5j, p. 168, line 
7) the following 

Lemma 5.4. 

G GL(2,Z) I tr(A[f/]) < |A|} < G GL(2,Z) | tr(iJ[f/]) < 

where we set H = ^ ^ for A = ^ . /n particular, 

#{f/ G GL(2,Z) I tr(A[[/]) < |A|} < 
z/A is Minkow ski-reduced. 

Therefore, the second sum on the right-hand side of (15. 4p is 

(7GGL(2,Z),tr(A[(7])<|A| 
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Collecting the above estimates, and noting ||A^C|| = A^^|ciC2|, from (I5.4p we obtain 



(5.6) 



E 

i7GGL(2,Z) 



DmodAfCA 



, fc 3 1 



< |T|^-tA^-t+e|ci|t|c2|-i+^(A^C2,t4)^ 



-2k+3 I 



A\2~-4+^ + Ar-'=+2m(A)^max(l, 2-3- 



Similarly to fl4.4p . we see that ^4 = T[v], where v is the vector consisting of the second 
column of V. Using this fact, 1^41 = |T|(ciC2)~^ and {Nc2,T[v])2 < A^s (c2, T[f]) 2 , we find 
that the right-hand side of fl5.6p is 



+N-''+^m{T[Vdi3.g{ci,C2)-'U{']Y ■ max(l, |r|(ciC2)-')^ 

Therefore, 
(5.7) 



|T|2-4--(CiC2) 



E E E 

0<cilc2 VGGL(2,Z)/P(ci/C2) C/eGL(2,; 



D mod NCA 



E E {W^l(Cl,C2,V^) + 1^2(Cl,C2,y)} 

0<ci|c2 yeGL(2,Z)/P{ci/c2) 



where 



H^l(Ci,C2,y) =|T|t-fiV-2'=+3+-cr'+'+-C2"'"^'''(c2,TH)5|T|t-3+- 



W^2(ci, C2, =|T|MiV-^'+2+-cr'+='+^C2 '+^^'(C2, TH)^m(T[l^diag(ci, C2)-^f/r^])^ 



X max(l, |T|(ciC2)"^)^+^ • |T|^-t-^ 



The form of W^2(ci, C2, is quite similar to the right-hand side of Kitaoka's Lemma 3 ([5], 
p. 169). Therefore, by the same argument as in p. 170 of Kitaoka |5], we obtain 



(5.8) 

On the other hand 



0<ci|c2 yeGL(2,Z)/P(ci/c2) 



(5.9) 



E E W^i(ci,C2,l^) 

0<ci|c2 VeGL(2,Z)/P(ci/c2) 



|2^|A:-l+ejY3-2fc+e ^ ^- 
0<ci|c2 



E (^2,TH) 

yeGL(2,z)/p{ci/c2) 



To evaluate this double sum, here we quote one more result of Kitaoka. For G = {gij) G A*, 
set e{G) := gcd^gu, g22,2gi2). Let ~ | ]b,dEZ, {b,d) = ij-. For a positive integer 

n, define 5'(n) = 5/ ~, where we denote ~ if there exists a w G Z such that 

fw, n) = 1 and | ^ j = w f j (mod n). 
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Lemma 5.5 (Kitaoka [5], p. 154, Proposition 2). For any P G A*, we have 

xeS{n) 

Applying this lemma to P[x] = T[v], n = c^jcx. Then 

yeGL(2,Z)/P(c2/ci) V'GGL(2,Z)/P(c2/ci) 

«chc2/Ci)^+^(c2/Ci,rH)^ 
< Cf(c2/Ci)5+^, 

where the second inequality follows from the lemma and the fact e(T) <t^ = T\p\. Therefore 
we have 

0<ci|c2 yeGL(2,Z)/P(c2/ci) 

oo oo 

0<Ci|c2 Cl=lC3 = l 

where we put C2 = C1C3. This is convergent when > 3. Then from fl5.9p we have 
(5.10) E E l^i(ci,C2,l^) < |T|'=-i+^Ar='-2fc+._ 

0<C2|ci VeGL2(Z)/P(c2/ci) 

Substituting (15. 8 p and (I5.10p into the right-hand side of (15. 7p . we now obtain 
Proposition 5.6. If k>3, then 

Next, we make use of (15. 3p (b) as well as (a) and (c) to deduce another estimate of S2. This 
time, instead of A, we use A' = T[Vdiag(A^Ci, A^C2)"^f/i]. Then \P\ x \A'\, trP x tr(A'[t/]), 
and hence the sums in the curly parentheses on the right-hand side of (15. 4p are replaced by 

(/GGL(2,Z),tr(A'[[/])<l l/eGL(2,Z),tr{A'[l/])<|A'| 

+ E |A'|M(tr(A'[[/]))V, 

C/eGL(2,Z),otherwise 

for which (15. 5 p can be directly applied. Therefore 

E E E E 

0<ci|c2 yGGL(2,Z)/P(ci/c2) C/GGL(2,Z) DmodAfCA 

« E E |T|i-t||ArC||-i(Arci)2(Arc2)^+^(Arc2,t4)^ 

0<ci|c2 yGGL{2,Z)/P(ci/c2) 

X m(A7max(l, | 

< jV-2fc+f+£|2-|fe-2+£ ^ ^ 

0<ci|c2 VGGL(2,Z)/P(ci/c2) 



X (A^C2,TH)2max(l,A^-^|T|(ciC2 



Rl + i?2, 
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say, where Ri denotes the part with (ciC2)^ > N^^\T\, and R2 the remaining part. Then 

0<ci|c2 yeGL{2,Z)/P(ci/c2) 

(ciC2)2<iV-4|Tl 

o<ci\c2 yeGL(2,z)/p(ci/c2) 

(ciC2)2<JV-4|Tl 

«iV-i+-|T|t-f+- Y cfcj^' (^y^\c2/cue{T))l 

(ciC2)2<]V-4|T| 

by Lemma [5751 Putting C2 = rici, we obtain 

00 

ci=l n<|T|V2(ciAr)-2 

The last sum is 0(e(T)^(|T|5(ciA^)~^)^+'^) as is shown in p. 170 of Kitaoka [5]. Hence 

00 

(5.11) i?2 < A^-'+^|T|t-3+-e(T)^ Yl cr'^' ^ iV-2+^|T|t-i+^ 

Cl=l 

The remaining part -Ri can be treated similarly, and it is estimated by exactly the same 
right-hand side as that of f l5.1ip . Therefore we now obtain 

Proposition 5.7. 

6. Completion of the proof of Theorem 1.1 
From (12. lip . Proposition 13.21 Proposition 14.31 and Proposition 15.61 we have 
(6.1) ^Q,7v(T) = #{f/ G GL2(Z) I UQ'U = T} 

+ 0(A^5~''|T|^~5 + A^2-fe+£|j.||-i+e _^ ^3_2fc+e|y|fc-l+e^| 

for A; > 3. This implies assertions (i) and (ii) of Theorem 11.11 On the other hand, from 
(12. lip . Proposition 13.21 Proposition 14.51 and proposition 15.71 assertion (iii) follows. The 
proof of Theorem 11.11 is thus complete. 
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